NLG Evaluation: Let’s open up the box

There is a spectrum of possible shared tasks that can be used to compare NLG systems and from which we can learn. A lot depends on how we set up the rules of these games. We argue that the most useful games are not necessarily the easiest ones to play. The Lure of End-to-End Evaluation Mellish and Dale (1998) discuss a number of different approaches to NLG system evaluation that had been used by 1998. Systems can be evaluated, for instance, in terms of accuracy, fluency or in their ability to support a human task. Independent of this is the question as to whether evaluation is black box or glass box, according to whether it results in an assessment only of the complete system or also of its contributing parts. End-to-end evaluation is black box evaluation of complete NLG systems. It involves presenting systems with “naturally occurring” data and evaluating the language produced (according to accuracy, fluency, etc.). End-to-end evaluation is a tempting way to start doing NLG evaluation, because it imposes minimal constraints on the structure of the systems. Therefore as many people as possible can take part. This is important, because at the beginning critical mass is needed for things to “take off”. The Dangers of End-to-End Evaluation Unfortunately there are dangers in using an end-toend task as the basis of comparative NLG system evaluation: • Danger of overfitting the task. The best systems may have little to say about language in general, but may encode elaborate stimulus-response type structures that work for this task only. • Lack of generalisability. The best systems may have nothing to say about other NLG tasks. Or the way that systems are presented/ compared may prevent researchers in nearby areas from seeing the relevance of the techniques. So you may actually end up attracting fewer interested people.